Measuring effectiveness of electronic medical records systems: Towards building a composite index for benchmarking hospitals

INTRODUCTION Many hospitals are currently in the process of developing and implementing electronic medical records (EMR) systems. This is a critical time for developing a framework that can measure and allow for comparison the effectiveness of EMR systems across hospitals that have implemented these systems. The motivation for this study comes from the realization that there is limited research on the understanding of the effectiveness of EMR systems, and a lack of appropriate reference theoretical framework for measuring the effectiveness of EMR systems. In this paper, we propose a conceptual framework for generating a composite index (CI) for measuring the effectiveness of EMR systems in hospitals. METHODS Data used to test the framework and associated research objectives were derived from a cross-sectional survey of five stakeholders of EMR systems including chief medical officers, chief nursing officers, chief information officers, doctors and nurses in 20 Japanese hospitals. Using statistical means of standardization and principal component analysis (PCA) procedure, CI was developed by summing up the scores of four dimensions-system quality, information quality, use and user satisfaction. The process included formulating items for each dimension, condensing the data into factors relevant to the dimension and calculating the CI by summing up the product of each dimension with its respective principal component score coefficient. RESULTS The Cronbach's alpha for the four dimensions used in developing CI was .843. Validation of CI revealed that it was correlated to internal dimensions (system quality, R=.828; information quality, R=.909; use, R=.969; and user satisfaction, R=.679) and to external factors (JAHIS level, R=.832 and patient safety culture, R=.585). These results suggest that CI could be a reliable and valid measure of the effectiveness of EMR systems in the responding hospitals. On benchmarking of hospitals, 30.0% (6/20) of the responding hospitals performed less than satisfactory on CI and that majority of the hospitals performed poorly on user satisfaction. CONCLUSIONS CI has provided a standard way, through quantitative means, of measuring, comparing and categorizing the effectiveness of EMR systems in hospitals. CI can be a powerful tool for benchmarking the effectiveness of EMR systems in hospitals in ways that can guide hospitals in computerization process as well as benchmark their systems against other hospitals.